[AMD][MI355X] Add DSv4 FP4 agentic MTP recipe with DPA tuning - #2254
[AMD][MI355X] Add DSv4 FP4 agentic MTP recipe with DPA tuning#2254seungrokj wants to merge 65 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 如需进行 PR 验证,请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
| dsv4-fp4-mi355x-sglang-agentic-hicache: | ||
| image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 | ||
| model: deepseek-ai/DeepSeek-V4-Pro | ||
| model-prefix: dsv4 | ||
| runner: cluster:mi355x-amds | ||
| precision: fp4 | ||
| framework: sglang | ||
| multinode: false | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8] } | ||
| - { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 48, 64] } | ||
| - { tp: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64] } |
There was a problem hiding this comment.
🟡 This PR accidentally leaves a second, byte-for-byte identical dsv4-fp4-mi355x-sglang-agentic-hicache: block in configs/amd-master.yaml (lines 2468-2482), duplicating the pre-existing entry directly above it. It's a copy-paste artifact from using the hicache block as a template for the new dsv4-fp4-mi355x-sglang-agentic-mtp entry — please delete the duplicate block.
Extended reasoning...
The diff adds a second top-level dsv4-fp4-mi355x-sglang-agentic-hicache: mapping key immediately below the pre-existing one, right before the new dsv4-fp4-mi355x-sglang-agentic-mtp: entry that this PR is actually meant to introduce. In the resulting file this shows up as two adjacent, byte-for-byte identical blocks: the pre-existing entry at lines 2452-2466, followed by the newly-added duplicate at lines 2468-2482.
Code path: configs/amd-master.yaml is loaded via yaml.safe_load in load_config_files() (utils/matrix_logic/validation.py:919). PyYAML's default loader does not error on duplicate mapping keys within a single document — it silently keeps only the last occurrence. Since both dsv4-fp4-mi355x-sglang-agentic-hicache blocks are identical, the "last one wins" collapse produces the exact same effective config as if the duplicate had never been added, so there is no behavioral difference today.
Why the repo's own duplicate-key guard doesn't catch it: load_config_files() does have a duplicate-key check at validation.py:930-936, but it only detects a key being duplicated across different config files (it intersects config_data.keys() from the current file against all_config_data.keys() accumulated from previously-loaded files). By the time that check runs, yaml.safe_load has already silently collapsed the intra-file duplicate down to one key, so the duplication within amd-master.yaml itself is invisible to it. There's also no .yamllint config in the repo that would otherwise flag duplicate keys at CI time.
Impact: Purely cosmetic/dead config today, since the two blocks are identical and the parser deterministically resolves to one value. However, this is exactly the kind of artifact that causes real confusion later: if a future PR edits one of the two blocks (e.g. bumping the image tag or tweaking the search space) without noticing the sibling duplicate, the edit could silently have no effect (if the untouched copy happens to load last) or silently take effect while looking like a no-op in a diff review. It's clearly unintentional — the PR description says it's adding only the new -mtp entry, and the mechanism is obviously that the hicache block was copied as a scaffold for the new entry and the copy of the original block was never deleted.
Proof / walkthrough:
- Before this PR:
dsv4-fp4-mi355x-sglang-agentic-hicache:appears once, at (pre-PR) line ~2465. - This PR's diff inserts 32 new lines directly after it, starting with
dsv4-fp4-mi355x-sglang-agentic-hicache:again (identical image, model, scenarios, search-space), followed by the newdsv4-fp4-mi355x-sglang-agentic-mtp:block. - Post-PR,
grep -n '^dsv4-fp4-mi355x-sglang-agentic-hicache:' configs/amd-master.yamlreturns two hits (lines 2452 and 2468 in the final file). - Running the file through
yaml.safe_loadcollapses these to a single key with the (identical) value of the second occurrence — no functional change today, but a landmine for future edits and a source of reviewer confusion.
Fix: Delete the duplicated dsv4-fp4-mi355x-sglang-agentic-hicache: block introduced by this PR (lines 2468-2482), leaving the original entry and the new dsv4-fp4-mi355x-sglang-agentic-mtp: entry intact.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29548395719 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29548395719 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29555826772 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29562088440 |
d895957 to
4ddd584
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29635054984 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29635071153 |
5324d24 to
3040b2a
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29678199226 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29678318533 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29679892938 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30591362551 |
…s-python >=3.11) installs on the mi355x sglang-rocm image (default python3 is 3.10) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30604232312 |
…or prefix-affinity across DP ranks consistent_hashing keyed on per-request-unique x-correlation-id scattered a session's turns across DP ranks, so the radix cache could not be reused (GPU cache hit ~67% vs ~97% theoretical at conc=64). Switch to cache_aware (radix-tree longest-prefix match per DP rank) and drop the forced correlation-id routing key so routing follows request content. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30613864535 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30613919909 |
…nto amd/agentx_dsv4_sgl_mtp_0717
…0260731 and restore full conc curve Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iAnalysisAI/InferenceX into amd/agentx_dsv4_sgl_mtp_0717
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30687991480 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30687991480 |
Long-context agentic warmup (ISL p95 ~433k tokens) with MTP exhausted HBM (HSA_STATUS_ERROR_OUT_OF_RESOURCES, 0 MB free) in the DSv4 attention indexer's transient paged-MQA logits buffer, crashing the scheduler. Lower MEM_FRACTION_STATIC 0.85->0.80 for headroom and cap the prefill chunk at 16384 to bound that per-chunk allocation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30702329471 |
…op prefill cap Revert the earlier OOM workaround: the passing run 29651998085 (image v0.5.15.post1-20260714) used mem-fraction 0.90 and a larger prefill chunk yet did not OOM, so lowering them was not the right lever. Restore mem-fraction 0.9 and remove the 16384 prefill-chunk ceiling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30729040169 |
Align the recipe with passing run 29651998085 (v0.5.15.post1) to isolate the v0.5.16 indexer prefill-workspace OOM: chunked-prefill 4096->8192, swap --enable-dp-attention-local-control-broadcast back to --enable-prefill-delayer, router policy cache_aware->consistent_hashing, and restore the SMG routing-key env. mem-fraction already 0.9. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30730205836 |
…row to c48 Revert to the last known-good image (v0.5.15.post1-20260714) and run c48 only, to confirm the OOM regression rides with the v0.5.16 image rather than the recipe. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…to amd/agentx_dsv4_sgl_mtp_0717
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30731238794 |
…,32,48] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30732631107 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30732835926 |
Summary
dsv4-fp4-mi355x-sglang-agentic-mtpconfig (separate from hicache)dsv4_fp4_mi355x_sglang_mtp.shwith EAGLE spec decoding (3 steps, topk 1, 4 draft tokens)--enable-prefill-delayer,GPU_MAX_HW_QUEUES=5, chunked prefill8192*TPMAX_RUNNING_REQUESTS = 2*CONC,CUDA_GRAPH_MAX_BS = CONC(cap 128)MEM_FRACTION_STATICreduced by 0.10 when spec decoding is activelmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260714Test plan
🤖 Generated with Claude Code